Deploy YOLOv8 on NVIDIA Jetson using TensorRT and DeepStream SDK

This guide explains how to deploy a trained AI model into NVIDIA Jetson Platform and perform inference using TensorRT and DeepStream SDK. Here we use TensorRT to maximize the inference performance on the Jetson platform.

For YOLOv8 to work together with DeepStream, we are using this DeepStram-YOLO repository and it supports different versions of DeepStream. So make sure to use the correct version of JetPack according to the correct version of DeepStream.

To verify this wiki, we have installed DeepStream SDK 6.1.1 on a JetPack 5.0.2 system running on Jetson AGX Orin 32GB H01 Kit. At the time of writing this wiki, DeepStream-YOLO only supports DeepStream SDK 6.1.1 which means you need to flash JetPack 5.0.2 on the device. However, Orin NX (reComputer J4012) only supports JetPack 5.1 and above and if you want to test this on Orin NX, you need to emulate Orin NX on the Jetson AGX Orin Dev Kit or AGX Orin 32GB H01 Kit with JetPack 5.0.2. Refer to this detailed guide about emulation to learn more.

Now you need to make sure that the Jetson device is flashed with a JetPack system including SDK components such as CUDA, TensorRT, cuDNN and more. You can either use NVIDIA SDK Manager or command-line to flash JetPack to the device.

For Seeed Jetson-powered devices flashing guides, please refer to the below links:

There are multiple ways of installing DeepStream to the Jetson device. You can follow this guide to learn more. However, we recommend you to install DeepStream via the SDK Manager because it can guarantee for a successful and easy installation.

Note: torch and torchvision are excluded for now because they will be installed later.

If the installer complains about outdated python-dateutil package, upgrade it by

We cannot install PyTorch and Torchvision from pip because they are not compatible to run on Jetson platform which is based on ARM aarch64 architecture. Therefore we need to manually install pre-built PyTorch pip wheel and compile/ install Torchvision from source.

Visit this page to access all the PyTorch and Torchvision links.

Here are some of the versions supported by JetPack 5.0 and above.

PyTorch v1.11.0

Supported by JetPack 5.0 (L4T R34.1.0) / JetPack 5.0.1 (L4T R34.1.1) / JetPack 5.0.2 (L4T R35.1.0) with Python 3.8

file_name: torch-1.11.0-cp38-cp38-linux_aarch64.whl URL: https://nvidia.box.com/shared/static/ssf2v7pf5i245fk4i0q926hy4imzs2ph.whl

PyTorch v1.12.0

Supported by JetPack 5.0 (L4T R34.1.0) / JetPack 5.0.1 (L4T R34.1.1) / JetPack 5.0.2 (L4T R35.1.0) with Python 3.8

file_name: torch-1.12.0a0+2c916ef.nv22.3-cp38-cp38-linux_aarch64.whl URL: https://developer.download.nvidia.com/compute/redist/jp/v50/pytorch/torch-1.12.0a0+2c916ef.nv22.3-cp38-cp38-linux_aarch64.whl

For example, here we are running JP5.0.2 and therefore we choose PyTorch v1.12.0

Here is a list of the corresponding torchvision version that you need to install according to the PyTorch version:

If you want a more detailed list, please check this link.

NOTE: You can use your custom model, but it is important to keep the YOLO model reference (yolov8_) in your cfg and weights/wts filenames to generate the engine correctly.

Note: To change the inference size (defaut: 640)

The above result is running on Jetson AGX Orin 32GB H01 Kit with FP32 and YOLOv8s 640x640. We can see that the FPS is around 60 and that is not the true FPS because when we set type=2 under [sink0] in deepstream_app_config.txt file, the FPS is limited to the fps of the monitor and the monitor we used for this testing is a 60Hz monitor. However, if you change this value to type=1, you will be able to obtain the maximum FPS, but there will be no live detection output.

For the same video source and the same model as used above, after changing type=2 under [sink0], the below result can be obtained.

As you can see, we can get an fps of about 139 which relates to the real fps value.

If you want to use INT8 precision for inference, you need to follow the steps below

Step 3. For COCO dataset, download the val2017, extract, and move to DeepStream-Yolo folder

Step 4. Make a new directory for calibration images

Note: NVIDIA recommends at least 500 images to get a good accuracy. On this example, 1000 images are chosen to get better accuracy (more images = more accuracy). Higher INT8_CALIB_BATCH_SIZE values will result in more accuracy and faster calibration speed. Set it according to you GPU memory. You can set it from head -1000. For example, for 2000 images, head -2000. This process can take a long time.

From

To

Step 9. Before running the inference, set type=2 under [sink0] in deepstream_app_config.txt file as mentioned before to obtain the max fps performance.

Step 10. Run the inference

Here we get an FPS value of about 350!

Included in the samples directory is a command-line wrapper tool called trtexec. trtexec is a tool to use TensorRT without having to develop your own application. The trtexec tool has three main purposes:

Here we can use trtexec tool to quickly benchmark the models with different parameter. But first of all, you need to have an onnx model and we can genrate this onnx model by using ultralytics yolov8.

For example:

This will output performance results as follows along with a generated .engine file. By default it will convert ONNX to an TensorRT optimized file in FP32 precision and you can see the output as follows

Here we can take the mean latency as 7.2ms which translates to 139FPS. This is the same performance we got in the previous DeepStream demo.

However, if you want INT8 precision which offers better performance, you can execute the above command as follows

Here we can take the mean latency as 2.9ms which translates to 350FPS. This is the same performance we got in the previous DeepStream demo.

These are some of the performance benchmark results based on our testing on the reComputer J4012

Please submit any technical issue into our forum.